Method and system for video based image detection/identification analysis for fluid and visualization control

ABSTRACT

In a surgical system, a system controller executes a video signature identification and image control routine to maintain quality of a video image taken by a video camera located at a surgical site and provided on a video display. The system includes a video camera/light source handpiece for insertion into a patient body. A tool is inserted separately into the surgical site. Fluid input into the surgical site is provided by a liquid pump or by an insufflator. Video signals are analyzed and fluid input/output, fluid pressure, and/or tool operation is automatically controlled to maintain image quality of the surgical site without manual adjustments.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.61/340,805, filed Mar. 23, 2010, the disclosure of which is incorporatedherein by reference in its entirety.

FIELD OF THE INVENTION

This invention is directed to a method and system for analyzing videoimages to obtain an acceptable image of a surgical site for display anduse in a surgical procedure.

BACKGROUND OF THE INVENTION

In a typical arthroscopic surgical procedure, fluid is pumped into asurgical field to create a positive pressure in a joint to provide spaceto perform the procedure and to reduce bleeding by creating a tamponadeeffect. Fluid is pumped into and out of the joint to remove debris andblood from the surgical site. A video sensing device typically providesimages of the surgical site. A surgeon manually controls the inflow andoutflow of fluid from the surgical site to prevent debris, blood orother particulates from degrading the image that is necessary to performthe procedure. This manual task can be difficult and lead to additionaltime for conducting a surgery and potentially compromise the surgery ifthe images are not acceptable.

Prior art U.S. Patent Pub. No. 2008/0243054 A1 discloses a knownarrangement, as shown in prior art FIG. 1, for detecting anddistinguishing blood and debris disposed within a shoulder joint duringan arthroscopic procedure. The pressure in the shoulder joint, and theliquid flow from an irrigation system, is controlled to attempt toprovide good viewing with the arthroscope by keeping video camera imagesfree from blood and debris.

In prior art FIG. 1, the arrangement includes an operating controldevice 17 and a saline bag 10 connected to a liquid inflow pump 2 thatpressurizes feed tubing 13. The arrangement includes a pressure sensor 5joined to the feed tubing 13. The feed tubing 13 connects to anarthroscope 14 that includes a video camera 22. A shaver 15 connects toan outflow tubing 16 that is in fluid communication with an outflow pump3. During flow, the pressure drops all the way from the inflow liquidpump 2 to the outflow pump 3. In prior art FIG. 1, a video signalprocessor 23 is shown adjacent the operating control device 17.

In prior art FIG. 1, the detection of hemoglobin is obtained by use oflight reflected from the hemoglobin in blood disposed in the joint 1.The light reflected is that which is radiated by a light source (notillustrated) known in the art of endoscopy. The light is introduced in adesignated channel in the arthroscope 14 that is placed in the shoulder.The reflection of white light is detected by the video camera 22, whichis fitted on the arthroscope 14. The video signal that is output fromthe video camera 22 is provided to a video signal processor 23.

The video signal is composed of separate red, blue and green videosignal components that, in combination, compose the video colorinformation. The video signal from the camera 22 is fed to the signalprocessor 23 that divides every video line signal into 0.64 microsecondtime slots. This arrangement corresponds to 100 time slots for everyvideo signal line, where a picture frame is made up of 625 lines (PAL).The signal levels of red, blue and green are nearly the same for theimages common in the view field of the arthroscope, meaning that it isgenerally ranging from white to black. If the signal level of redis >20% of either the blue or green during a time slot, a score of oneis registered in a first frame memory in the signal processor 23. If thewhole image is red, 62,500 score points would be registered. Everypicture frame has its own score. The first frame memory in the signalprocessor 23 has a rolling register function that has the capacity ofscoring points from 10 frames. The frame memory is updated as everyframe is completed and delivered by the camera. At every new frame, thescore value of the oldest frame is discarded. A score sum for the tenframes is calculated every time a new frame is delivered by the camera,thus introducing an averaging function.

If, during a period of 10 frames, the score sum is >30,000, blood isconsidered present, and if the score sum is >70,000, much blood isconsidered present.

If the score sum is >30,000, the video signal processor 23 will signalto the operating control device 17 which will react by increasing theflow of the aspiration pump to a higher level to rinse the shoulder.This higher level is preselected in the menu of the pump system, and isin this case 300 ml/min. If the score sum is >70,000, the flow willincrease to 450 ml/min. When the blood detection determines that theincreased flow has rinsed the shoulder as the signal level has returnedto a low level, the aspiration pump will return to the intrinsic flow of150 ml/min after a timeout of 30 seconds. Also, to stop bleeding of aruptured blood vessel in the shoulder joint 1, the pressure in the jointis increased by a pressure control for the same time that flow iselevated. This pressure increase is predetermined in menu settings ofthe pump, and is in this example 40 mm Hg. Also other picture analysistechniques as known in the art could be used.

To detect debris, the signal processor 23 divides every video frame into128×128 pixel elements. Every such pixel has a signal level thatcorresponds to the whiteness of the object that is visualized by thecamera. This whiteness is nearly the same from video image frame tovideo image frame. The signal processor stores a value from 0 to 15 asthis intensity value of the video signal of each pixel in a second framememory. The pixel values are stored in a matrix fashion. For each videoimage frame 25 consecutive frame matrixes are stored. The second memoryin the signal processor has a rolling register function that rolls the25 frames in a first in-first out fashion. The second memory is updatedas every video image frame is completed and delivered by the camera. Asa new frame is developed by the camera, the oldest frame is discarded. Avariation in the pattern in the second stored matrix is detected by thesignal processing unit. This variation is identified as pixelintensities that are recognized as moving from one adjacent pixel toanother in an identifiable fashion. As every pixel has a location in thematrix that corresponds to the physical image, a movement of intensityin the matrix location from image frame to image frame is a movement inrelation to the surrounding, of a single object, in this case debristhat float in the shoulder joint. Movement can be in any direction inthe matrix. If 10 such movements are detected during one frame, a firstscore value is incremented by one in a memory cell representing a firstscore value. This score value is incremented for each detection, and isdecremented down to 0 for every frame there is no such detection. Ifthere are over 500 detections in one frame, the camera is moved, and noscore values are given. Also, other picture analysis techniques as knownin the art could be used.

Every second a frame matrix is stored in a third frame memory. Thismemory also has a rolling register function that rolls the 25 frames ina first in-first out fashion. If predominant consistently low signallevels are occurring in the third frame memory, dark areas areidentified. If these dark areas are elevated to a consistent signal >25%level over a time of 5 seconds, homogeneous debris is identified asbeing present in the shoulder joint. Such occurrence increases the valueof a second score value by 10. If there is no such occurrence, thissecond score value will be decremented by 10 down to 0.

If either the first or second score values are >50, debris is consideredpresent, and the video signal processor 23 will signal to the operatingcontrol device 17 which will react by increasing the flow of theaspiration pump to a higher level to rinse the shoulder. This higherlevel is preselected in the menu of the pump system, and is in this case300 ml/min. When the debris detection determines that the increased flowhas rinsed the shoulder as the score value has returned to <50, theaspiration pump will return to the intrinsic flow of 150 m./min after atimeout of 5 seconds, and both score values are reset.

The system described above, however, is limited to an operating controldevice 17 for a pump that only controls liquid inflow for thearthroscope 14 and liquid outflow from a separate shaver 15. In thissystem, there is no control of any functions other than the liquidinflow pump 2 and liquid outflow pump 3. Further, the prior art systemis limited to processing red, blue and green video signal components.Finally, the FIG. 1 system is limited to controlling flowpressure/inflow and outflow rates in order to obtain a desired videoimage.

SUMMARY OF THE INVENTION

The present invention provides an enhanced video image for a surgicalsite by determining one or more of a plurality of conditions thatdegrade the quality of at least portions of the video image of asurgical site, and automatically selecting from a plurality of systemcontrol arrangements the optimal functions to be controlled in order tomaximize the quality of the video image provided on a display for asurgeon during a medical procedure.

In one embodiment of the invention, a video based imagedetection/identification system for sensing fluid and for visualizationcontrol is provided with a surgical system including a cutting tool formanipulating tissue, a pump system for providing fluid, a light source,and a video sensing device for obtaining video signals of images at asurgical site for display. A system controller controls at least one ofthe cutting tool, the pump system, and the video sensing device tomaintain image quality of the images obtained by the video sensingdevice for display. The controller receives and processes video signalsfrom the video sensing device to determine video signatures thatcorrespond to specific conditions at a surgical site that interfere withimage quality. In response to an identified condition at the surgicalsite that degrades the video image, the controller controls the cuttingtool, pump system, light source and/or video sensor device so that theimages from the video signals return to an acceptable qualityautomatically, so as to free a user from the task of manuallycontrolling the devices to maintain a desired video image. The videosignatures determined by the controller include one or more of bubbles,debris and bleeders located at a surgical site as viewed by the videosensing device of an endoscopic system. Another video signaturedetermined by the controller determines the presence of debris locatedat a distal tip of a housing of a handpiece associated with a videosensing device.

In one embodiment, the invention is provided with a laparoscopic systemthat utilizes an insufflator as a pump to provide gas to a surgicalsite. In this embodiment, the video signatures are provided to detectsmoke, debris and other conditions.

In some embodiments, predetermined wavelengths of energy are providedfrom the light source to the surgical site. The energy for determiningthe conditions at the surgical site can be emitted at wavelengths thatare out of the visible light range.

In response to the various video signatures, the system controller ofone embodiment of the invention controls at least one of suctionpressure, a suction valve position, input pressure, an input flow valve,color imaging processing of the camera control unit, light sourceintensity, and color balance of the video sensing device to improve thevideo image for display.

By directly comparing video images for quality and directly operatingvarious devices in response to image quality issues, the inventionoperates as a closed system. More specifically, the quality of the imageis not obtained or improved by sensing pressure values or otherconditions that do not have a direct effect on the image. Instead, thecontroller acts as a closed system by receiving direct video informationas to the quality of an image and then controlling devices to maintainimage quality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a prior art arthroscopic system.

FIG. 2 depicts a block diagram of an integrated surgical control system,along with surgical tools disposed at a surgical site.

FIG. 3 is a flow chart showing steps for a video signatureidentification and image control routine that operates to provide anenhanced image of a surgical site.

FIG. 4 is a flow chart of a video image analysis subroutine foranalyzing video images from a surgical site.

FIG. 5 is a graph of a video signal stream, color signals, spatialfrequency and a Q scale.

FIG. 6 is a signature match image adjustment subroutine for controllingdevices to improve the video image.

FIG. 7 depicts a block diagram of a laparoscopic integrated surgicalcontrol system, along with surgical tools disposed at a surgical site.

FIG. 8 is a flow chart of a video image analysis subroutine foranalyzing video images from a surgical site.

FIG. 9 is a flow chart for a multi-band light source control imageanalysis subroutine.

FIG. 10 is a flow chart of a leak/clog subroutine of the invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 2 illustrates an integrated surgical control system 30 thatincludes a system controller 34 having a display panel 36 and an inputdevice 38. The system controller 34 is connected by a computer network40 to communicate with a plurality of surgical devices. The devices mayinclude a cutting tool 42, such as a surgical shaver, having a handpiece44 connected by a signal line 45 to a cutting tool controller 46 thatcommunicates with the system controller 34 over the network 40. Thesurgical control system 30 can be an arthroscopic surgical controlsystem.

The surgical control system 30 includes a pump system 50 connected tosuction tubing 52 for providing suction to a cannula 54 having a stopcock 55. In some embodiments, the pump system 50 includes a peristalticpump. While the cannula 54 is only shown as providing a suction flowpath, in some embodiments, the cannula 54 can be used to define a portalinto the surgical site into which other types of tools or devices can beplaced. Such devices include cutting tools, irrigation devices and othersurgical instruments. The pump system 50 communicates with the systemcontroller 34 via the network 40. While a single suction tubing 52 isshown connected to the pump system 50, in some embodiments a pluralityof suction tubings from different devices may connect to the pumpsystem.

The surgical control system 30 includes a wall suction unit 56 andsuction tubing 57 which connects the suction unit to the cutting tool 42via the cutting tool handpiece 44 to provide suction for removing debrisresulting from tool operation. The wall suction unit 56 includes anadjustment valve (not shown) for controlling the suction for tubing 57.In some embodiments, the valve is automatically adjusted by thecontroller 34 to automatically vary the pressure within tubing 57. Whilepump system 50 provides suction for tubing 52, in some embodiments thepump system 50 does not provide suction. Instead, wall suction unit 56provides suction to tubing 52 and additional tubing. The wall suctionunit 56 can have control valves to control suction separately for eachtubing connected thereto. In some embodiments, a portable suction unitis provided when a wall suction unit is not available.

The surgical control system 30 also includes a camera control unit (CCU)60 that communicates via the network 40 with the system controller 34.The camera control unit 60 connects via video signal line 62 to a videocamera 64 that is mounted onto or is integral with a video camera/lightsource handpiece 66 disposed within a cannula 67 providing access to asurgical site as shown in FIG. 2. The handpiece 66 is configured tooutput light to a surgical site. In some embodiments, a separateconnection 68 (shown in broken line in FIG. 2) is provided to transferthe video signal from the camera control unit 60 directly to the systemcontroller 34. In some embodiments, the handpiece 66 includes a C-mountendoscope having a focus ring and a camera head with a video camera thatis attached to the proximal end thereof.

A light source controller 70 communicates with the system controller 34via the network 40 and controls power to a light source 74 that provideslight to the handpiece 66 via light source optical fiber 72. In otherembodiments, the optical fiber 72 is replaced by a power line thatprovides power to a light source mounted in the handpiece 66. In someembodiments, the video camera/light source handpiece 66 has a videocamera (not shown) provided at the distal end of the handpiece, insteadof the video camera 64 located at a proximal end of the handpiece.

Irrigation tubing 76 connects the pump system 50 to the cannula 67 toprovide irrigation fluid to the handpiece 66. The fluid is output toirrigate surgical site 80. A stop cock or valve (not shown) controlsirrigation fluid into the cannula 67. The irrigation fluid follows apath between the inner wall of the cannula 67 and about the outerperiphery of the video camera/light source handpiece 66 to the surgicalsite 80 at the distal end of the cannula. The video camera/light sourcehandpiece 66 both projects light outwardly at the distal end thereof andsenses video images with the video camera 64.

Video display 78 shown in FIG. 2 communicates with the system controller34 over the network 40. The video display 78 displays the video imagetaken at a surgical site 80 by the video camera 64. In some embodiments,a video display signal line 82 (shown in dotted lines in FIG. 2) isprovided for directly connecting the camera control unit 60 to the videodisplay 78.

In FIG. 2, the handpieces 44, 66 and cannulas 54, 67 are oriented withinsurgical site 80 for conducting a surgical procedure.

In the surgical control system 30, the system controller 34 receivesinputs from the cutting tool controller 46, pump system 50, video cameracontrol unit 60 and the light source controller 70. The video display 78typically is a high definition LCD display whereat the image taken bythe video camera 64 of the handpiece 66 is displayed continuously.

The input device 38 of the system controller 34 in some embodiments is atouch screen, while in other embodiments the input device is a set ofcontact switches or the like. Further, the input device 38 may include avoice recognition system for processing voice commands to providecontrol signals to the system controller 34.

In one embodiment, video signals from the camera control unit 60 are,instead of being provided over the network 40, simply sent separatelyand wirelessly to the system controller 34 and to the video display 78.

While pump system 50 is shown in FIG. 2 as having one suction input andone irrigation output, it is contemplated that the pump system may havea plurality of suction inputs. Wall suction unit 56 with valve controlcan connect to the suction tubing 52 as well as other tubing, so thatthe pump system 50 only comprises an irrigation pump. A portable pumpsystem for providing irrigation is also contemplated.

Video Signature Identification and Image Control Routine

FIG. 3 is a flow chart representing a video signature identification andimage control routine 86 for the system controller 34 that controls thedevices of the surgical control system 30. The video signatureidentification and image control routine 86 begins at start 88. Fromstart 88, the video signature identification and image control routine86 advances to signal receiving step 90 wherein the system controller 34receives video signals from the camera control unit 60 and advances toimage analysis subroutine 92. In other embodiments, the camera controlunit 60 performs the image analysis. In some embodiments, the cameracontrol unit 60 is mounted with the camera 64 that is located at theproximal end of the video camera/light source handpiece 66.

At image analysis subroutine 92, the system controller 34 analyzes imagedata taken from the video signals for determining the presence of videosignatures and stores identifiers in response to matches as shown in theflow chart of FIG. 4 discussed in detail later herein.

Returning to FIG. 3, the image control routine 86 then advances fromimage analysis subroutine 92 to video signature match step 94. Atdecision step 94, system controller 34 looks for the presence of storedidentifiers corresponding to the various video signatures as determinedby and stored in the controller during operation of the image analysissubroutine 92.

In instances where the image analysis subroutine 92 does not find avideo signature that matches with a predetermined stored videosignature, no identifier is present at decision step 94, and theidentification and image control routine 86 advances to image datadecision step 96.

At decision step 96, at least a portion of the image data previouslyanalyzed at subroutine 92 is compared with optimal desired image dataproperties. If the analyzed image is considered appropriate, the routine86 advances from step 96 by returning to step 90 to receive new videosignals. The routine 86 is then repeated.

At step 96, when the video image is not at a maximized quality level,the system controller 34 advances to camera focus step 97. At step 97,focus of a lens (not shown) or the like provided with the videocamera/light source handpiece 66 is automatically controlled to maximizethe clarity of the video images. The system controller 34 then advancesto light adjustment step 98. At step 98, the output level of the lightsource 74 is either increased or decreased depending on the amount oflight necessary to maximize the quality of the video image taken byvideo camera 64 for display on the video display 78. After lightadjustment step 98, the image control routine 86 advances to colorbalance adjustment step 100.

At color balance adjustment step 100, the system controller 34 adjusts,if necessary, the color balance of the camera control unit 60 for thevideo images provided by the video camera 64. After color balanceadjustment step 100, the image control routine 86 advances to videosignal receiving step 90 and repeats the video signature identificationand image control routine 86.

In some embodiments, the color balance is adjusted internally by thecamera control unit 60 before a signal is output by the camera controlunit 60. In other embodiments, the system controller 34 performscalculations and provides an output to control the camera control unit60.

The order of adjustment steps 97, 98, 100 as shown in FIG. 3 is forpurposes of illustration only. The steps 97, 98, 100 may be performed inany order or performed essentially simultaneously by the systemcontroller 34.

Returning to video signature identifier present step 94, in an instancewherein a video signature identifier is stored in the controller 34, thecontroller advances the routine 86 to the signature match adjustmentsubroutine 106 as shown in the flow chart of FIG. 6 and discussed laterherein. Upon adjustment of the video image at subroutine 106, thecontroller 34 returns to step 90 of the video signature identificationand image control routine 86 as shown in FIG. 3, to receive new videosignals and to repeat execution of the video signature identificationand image control routine 86.

Image Analysis Subroutine

The image analysis subroutine 92 shown in FIG. 4 operates as follows.The subroutine 92 begins at start 148 and advances to process videoimage signals step 150 to obtain video signature information fromreceived video images. At step 150, the system controller 34 may conducta plurality of subcalculations or signature identification typeoperations on a video image, including comparisons with a sequence ofpreviously stored video images.

Image analysis subroutine 92 then advances to identify fast dispersionareas step 152. At step 152, processed video signals are compared withpreviously received and stored processed video signals to identify thepresence of video signatures for fast dispersion red areas (bleeders)within subsequent video images whereat blood is spreading at at least apredetermined minimum rate. If a bleeder is occurring, the subroutine 92advances to store identifier (ID) step 154. At step 154, the systemcontroller 34 then stores an identifier or identification codeidentifying the presence of a bleeder in the video images for latercontrol purposes.

The controller 34 then advances the subroutine 92 to find black/white(B/W) transitions decision step 156. If an identification of a videosignature for fast dispersion red areas is not matched at step 152, theimage analysis subroutine 92 advances directly to decision step 156.

At black/white transitions decision step 156, processed video signalsare compared with a sequence of previously processed video signals todetermine, over time, the presence of a video signature match for fastupwardly moving black/white transition, which correspond to upwardmovement of air bubbles within a liquid. The air bubbles can begenerated by operation of an RF cutting tool. If there is a signaturematch, or in some instances close to a complete signature match, thesubroutine 92 advances to storing identifier step 158.

At step 158, the controller 34 stores an identifier indicating thepresence of air bubbles in the video image. The controller 34 thenadvances the subroutine 92 to find objects at multiple distances step160. If there is not a signature match at step 156, the subroutine 92immediately advances to find objects at multiple distances step 160.

At find objects at multiple distances step 160, video images arecompared to determine the presence of video signatures for multiplesmall objects at different focal distances within a video image. Thissignature matching determines the presence of particulates in the fluidat the viewed surgical site 80. If the processed video images result ina video signature or signatures that match a stored video signature orsignatures for the presence of particulates, the video image analysissubroutine 92 advances to storing identifier (ID) step 162. Theparticulates can be debris generated by a burr or other cutting tool.

At storing identifier step 162, a particulate identifier or identifiercode is stored in the system controller 34. The subroutine 92 thenadvances to identify light dispersion step 164.

If there is not a video signature match for the identification ofparticulates at decision step 160, the subroutine 92 immediatelyadvances to identify light dispersion step 164.

At tissue blocking camera step 168, the controller 34 analyzes theprocessed video image signal to determine the presence of tissue or likematerial blocking the distal tip of the video camera/light sourcehandpiece 66 by comparing processed video images with a video signaturefor the presence of tissue blocking the distal end of the handpiece 66.

If a processed video image has a signature that matches with a storedvideo signature corresponding to the presence of tissue blocking thecamera view, the controller 34 advances the subroutine 92 to storeidentifier (ID) step 170.

At step 170, a camera tip blocked identifier is stored in the systemcontroller 34 and the subroutine 92 advances to return step 174. If ablocked distal tip for the camera/light source handpiece 66 is notdetected at step 168, the controller 34 immediately advances the imageanalysis subroutine 92 to return step 174.

The order of identification steps 152, 156, 160, 168 as shown in FIG. 4,is for purposes of illustration only. The steps 152, 156, 160, 168 insubroutine 92 can be performed in any order, or executed essentiallysimultaneously.

Upon completion of the identification of the various conditions that candegrade a video image taken at a surgical site 80, the controller 34advances from return step 174 of the video image analysis subroutine 92to the video signature identifier present step 94 of the video signatureidentification and image control routine 86 shown in FIG. 3.

With respect to video image signal processing or video image streamanalysis for the image analysis subroutine 92 discussed above,information can be extracted specifically by using color imagingprocessing algorithms. For instance, bleeders can be determined by usinga color image processing algorithm coupled with 2-D spatial informationfrom the sensed video image. For example, close pixels that look likeblood can be detected in the video RGB color space by using a simpledistance formula. The closer that the pixel distance is to the desiredpixel point in the color space, the more likely that the pixel belongsto a digital representation of a bleeder.

While determining and adjusting light output level and color balance isseparate from the image analysis subroutine 92 shown in FIG. 3, in someembodiments one or both of the light output level and the color balanceof the camera control unit 60 are provided with decision blocks as shownin FIG. 4 and compared with various video signatures in the imageanalysis subroutine 92.

In some embodiments, swelling/distention of joints can reduce quality ofa video image. This situation can be determined by video image analysisand pressure/fluid flow may be controlled to minimize the condition andany negative effect on the video image.

In image analysis subroutine 92, color image processing may also becoupled with other information and techniques, such as fast fouriertransform (FFT) spatial frequency information, image segmentation andweighting techniques to weight the different processing quantitativeindicators to make automated decisions about the existence ofobstructions, such as bleeders, along with particulates and bubbles.

FIG. 5 illustrates an embodiment wherein a video stream that representsa plurality of sequential video images is analyzed to determine thepresence of video signatures. In FIG. 5, color (RED color) and spatialfrequency (SF) are determined and shown over time for a stream of videosignals. A subjective image quality evaluation provides scaled scores asshown over time on a Q scale as illustrated in FIG. 5. The Q scalevalues can be used to determine the presence of a video signature orother anomaly. Low spatial frequency values may indicate the presence ofblood and high spatial frequencies may indicate the presence of debris.The presence of bubbles may have a unique signature spatial frequency.In response to a video signature match or the like, the controller 34then stores a predetermined identifier as discussed above.

Additional image analysis techniques are disclosed in U.S. Patent Pub.2008/0243054, the disclosure of which is hereby incorporated byreference. The '054 patent publication discloses an arthroscopic videocamera and video signal processor.

Signature Match Image Adjustment Subroutine

If, at the video signature identifier present decision step 94 in FIG.3, the controller 34 finds at least one stored identifier present, thecontroller advances to signature match adjustment subroutine 106.Detailed operation of the signature match image adjustment subroutine106 is illustrated in FIG. 6.

From start 208 in FIG. 6, the controller 34 advances the subroutine 106to control inflow pressure flow and/or flow rate control step 210.

At control step 210, the system controller 34 obtains information on theinflow pressure from a sensor (not shown) associated with the irrigationtubing 76 from the pump system 50 or from a sensor disposed in thehandpiece 66. Further, in some embodiments the system controller 34 isprovided with gas or liquid pressure sensed within the surgical site 80,as well as the flow rate through return suction tubing 52, 57 and thepressure values thereat. Depending on the measured pressure/flowconditions, and especially the type of condition at the surgical site 80determined by the matched video signature(s) and provided by the storedidentifiers, the controller 34 adjusts inflow pressure and/or flow rateprovided to the surgical site at step 210 to improve the quality of thevideo image taken by video camera 64. The controller 34 then advancesthe signature match image adjustment subroutine 106 to control outflowpressure and/or flow rate step 212.

At step 212, depending on the video signatures matched, the systemcontroller 34 selectively controls the outflow pressure and/or flow ratethrough the suction tubing 52, 57. The outflow pressure/flow ratecontrol is dependent in part on the input pressure/flow rate values, andthe type of identifiers. For example, in the case of a quantity ofincreasing blood areas detected by a video signature and provided withan identifier at step 154 as shown in FIG. 4, the inflow/outflow andpressure values can be operated in a manner to flush blood from thesurgical site 80 in a timely and effective manner. In some embodiments,pulsing of irrigation fluid entering a surgical site 80 removes theblood and provides a quality video image. The position of the stop cock55 of the cannula 54 or the valve of wall suction unit 56 canautomatically be adjusted by controller 34 to maximize the video image.

While steps 210, 212 are illustrated as separate steps in FIG. 6 forconvenience, in some embodiments the steps 210, 212 represent a singlestep as inflow/outflow control of fluid acts as a single response tocorrect a sensed undesired condition or plurality of undesiredconditions identified by a comparison with one or more video signaturesfrom subroutine 92. The controller 34 then advances the subroutine 106to camera color balance step 214.

Depending on the properties of the one or more video signature(s) thatare determined, selective control of lens focus for the video camera isprovided at camera focus step 213. In some instances, the systemcontroller 34 takes no action at step 213 and advances to step 214.

Depending on the properties of the one or more video signature(s) thatare matched or determined, selective control of camera color balanceoccurs in some embodiments at camera color balance step 214. In someinstances, especially depending on the type of video signatureidentifier obtained by the system controller 34, the controller takes noaction at step 214 and advances to light source output control levelstep 216.

At light output control level step 216, the controller 34 signals thelight source controller 70 for operating the light source 74 to outputmore or less light, as necessary, to maximize the video image output bythe camera control unit 60. More specifically, the system controller 34calculates the need for changes or adjustments in the light outputlevel, and then the controller 34 provides signals to the light sourcecontroller 70 to carry out the light output level changes. In someinstances, the light output level does not change as video image qualitywith respect to the light level is already maximized.

In some embodiments, the image adjustment subroutine 106 then advancesto step 218. Depending on the video signature identifiers that have beendetermined and stored, the controller 34 may automatically control, forexample, operation of the cutting tool 42 (on/off state orrotational/reciprocating speed) and also may control the inflowpressure/flow rate and/or outflow pressure/flow rate essentiallysimultaneously to unclog the cutting tool 42, or to otherwise improvethe video image taken of the surgical site 80 by camera 64. In someembodiments, a pulsing pressure and/or liquid flow is also applied toclean or unplug material at or near a cutting tool. In some embodiments,if bubbles are detected by image analysis, an RF cutting tool thatpresumably is generating the bubbles is controlled by the controller 34to operate in a different power mode or powered off to minimize theformation of bubbles.

In instances where the cutting tool 42 is not in operation, or noidentifier requires an action thereby, control of the cutting tool doesnot occur, and the image adjustment subroutine 104 simply advances thesubroutine 106 to control of camera tip cleaning step 220. In someembodiments, cutting tool control step 218 is not provided in the imageadjustment subroutine 106.

At step 220, if the controller 34 stores a camera cleaning identifierthat corresponds to a video signature caused by material or tissuedisposed at the distal end of the video camera/light source handpiece 66shown in FIG. 2, then the controller 34 controls an apparatus known inthe art to clean the distal tip of the handpiece 66 to obtain a desiredimproved image.

In some embodiments, the inflow of irrigation liquid is redirected by astructure adjacent or near the distal end of the handpiece to remove thetissue. In other embodiments, when a tip cleaning identifier isprovided, the other control steps, such as camera color balance andlight output level control are by-passed. The controller 34 advances theimage adjustment subroutine 106 directly to tip cleaning step 220 andthen to return step 222.

At return step 222, the system controller 34 returns to the videosignature identification routine 86 shown in FIG. 3 to receive videosignals at step 90 and repeat execution of the routine 86 to againdetermine the presence of video signature matches.

In some embodiments, the operating status of output devices or tools areeither monitored by or input to the system controller 34. In response tothe status of the devices or tools, the controller 34 can controlpressure and/or suction values, as well as control tool operation, tomaintain a quality image free from video signatures. For example, insome embodiments the system controller 34 provides output signals todrive the various cutting devices or receives input signals when a toolis manually operated. Thus, the system controller 34 is capable ofcontrolling inflow pressure flow and/or flow rate, as well as outflowsuction and/or flow rate, depending on the tools being operated. Thisarrangement enables the system controller 34 to initially prevent thedegradation of picture quality and thus avoid requiring a later actionin response to a video signature to improve the picture quality.

For example, when a burr or other cutting device is operating, thesystem controller 34 can immediately control suction and/or irrigationto remove debris before video image quality is degraded.

Laparoscopic System

The surgical control system 30 illustrated in FIG. 7 generallycorresponds to the system illustrated in FIG. 2 (like numbers utilizedfor the same parts), except the system is a laparoscopic surgicalsystem.

The surgical system 30 includes an insufflator unit 224 having airsupply tubing 226 for supplying air, preferably carbon dioxide, to atrocar 267 that receives the video camera/light source handpiece 66.Thus rather than supplying irrigation to surgical site 80, the trocar267 receives fluid, such as air or CO₂, between an inner wall of thetrocar and the periphery of the handpiece 66 that flows outwardly fromthe distal end of the trocar to expand a peritoneal cavity to enableaccess to a surgical site 80.

The pump system 50 and wall suction unit 56 shown in FIG. 7 connect to asuction/irrigation tool 228. The pump system 50 provides fluid to thetool 228 via irrigation tubing 229. The wall suction unit 56 providessuction to the suction/irrigation tool 228 via suction tubing 230. Thesuction/irrigation tool 228 selectively irrigates and suctions thesurgical site 80. In some embodiments, the wall suction unit 56 isreplaced by a portable suction unit.

The cutting tool 42 shown in FIG. 7 includes the handpiece 44 receivingan electrode 232. In some embodiments, the cutting tool 42 typically isan electrocautery device with the electrode 232 being capable of cuttingtissue and a coagulation function.

In some embodiments, the video camera/light source handpiece 66comprises an endoscope having an eyepiece that is joined at a proximalend to a camera head having a coupler.

The arrangement shown in FIG. 7 operates in a similar manner to thearrangement shown in FIG. 2. The suction/irrigation tool 228 typicallyis only manually controlled for irrigating surgical site 80 and forwithdrawing irrigation fluid as necessary.

The system controller 34 and network 40 communicate with and operate thevarious devices in essentially the same manner as shown in FIG. 3 anddiscussed above.

FIG. 8 shows a video image analysis subroutine 233 similar to thearrangement described above with respect to FIG. 4 (same numbers performthe same function).

In the video image analysis subroutine 233, steps 150, 152, 154 functionin the same manner as set forth above. After comparing video signaturesfor fast dispersion areas (bleeders) at step 152, and if found, storinga bleeder identifier at step 154, the controller 34 advances the videoimage analysis subroutine 233 to light dispersion identification step234.

At light dispersion identification step 234, the system controller 34operates to compare a sequence of processed video signals withpredetermined and stored light dispersion video signatures to determinethe presence of smoke. If there is a signature match between theprocessed video signal and a stored video signature for the presence ofsmoke, the controller 34 advances the subroutine 92 to store identifier(ID) step 235.

At store identifier step 235, a smoke presence identifier is stored inthe system controller 34. Then, the controller 34 advances thesubroutine 233 to tissue blocking camera tip decision step 168.

If a video signature for the presence of smoke is not identified by thecontroller 34 at step 235, the controller 34 immediately advances thesubroutine 233 to decision step 168.

At tissue blocking camera tip decision step 168, besides the possibilityof tissue on the lens of the camera/light source handpiece 66, thepresence of blood or the like on a lens and thus blocking the image forthe video camera 64 can be determined as a video signature match. If amatch is formed, the controller 34 advances to store identifier (ID)step 170 and thus stores an identifier.

After store identifier step 170, the controller 34 advances to return174 and returns to the video signature identification and image controlroutine 86 shown in FIG. 3. If there is no detection of tissue blockingof the camera 64 at decision step 168, the controller 34 immediatelyadvances to return step 174 and returns to routine 86.

The order of steps 152, 234, 168 in subroutine 233 is provided forpurposes of illustration only. The steps can be performed in any order,or essentially simultaneously.

In the laparoscopic surgical system, the signature match adjustmentsubroutine therefore is similar to the signature match adjustmentsubroutine 106 shown in FIG. 6. A main difference is that the control ofpressure steps 210, 212 are limited. In the laparoscopic surgicalsystem, in the event that smoke is detected by the video images from thesurgical site, the suction/irrigation tool 228 can automatically operateto remove smoke with the wall suction unit 56 via tubing 230. At thesame time, the insufflator unit 224 provides additional gas via thehandpiece 66 to the surgical site 80 for preventing the peritonealcavity from collapsing. Further, power to the electrode 232 of anelectrocautery device can be reduced or interrupted, if necessary, tolimit the production of additional smoke.

With respect to both arthroscopic and laparoscopic surgical systems, thecause of bleeding is less certain than other conditions resulting indegraded image quality. Therefore, in some embodiments, when a bleederis detected and none of the tool devices are operating, the systemcontroller 34 determines the tool device or devices that were mostrecently operated. The system controller 34 can utilize this informationto assist in determining what operations of fluid input/output, fluidpressure, or even which of plural fluid input/output devices to selectfor removing the bleeder from the video image.

Multi-Band Light Source Control Image Analysis Subroutine

Multi-band light source control image analysis subroutine 240 shown inFIG. 9 includes steps 250, 252, 254. The steps 250, 252, 254 are asubstitute for steps 150, 152, 154 in the video image analysissubroutine 92 shown in FIG. 4 and discussed above.

In multi-band light source control image analysis subroutine 240, thelight source controller 70 additionally controls the light source 74 toperiodically strobe or pulse the light source 74 to output additionalnon-visible near-infrared (IR) light at predetermined intervals. Sincethe additional near-infrared light is outside of the visible spectrum,the additional light is not viewable by a surgeon. Further, one or morebands or wavelengths of non-visible light can be provided at short timeintervals, such as milliseconds. Finally, in some embodiments, variouswavelengths of non-visible light are output simultaneously.

Since hemoglobin absorbs light in the 800 nm-1,000 nm (near-infrared)range, blood is visible as distinct dark points when reflected near-IRlight images are received by video camera 64. The dark points providemore detailed information for an image processing algorithm to determinethe presence of blood in an image, as compared to analyzing colors.Thus, instead of identification of fast dispersion areas at step 152 inFIG. 4 as discussed above, pulsed or strobed near-infrared signals, notviewable by the surgeon, are included with visible light and utilized tobetter determine the presence of fast dispersion red areas correspondingto bleeders within the video image obtained by the video camera 64.

To execute the multi-band light source control image analysis subroutine240 shown in FIG. 9, the light source 74 is controlled by light sourcecontroller 70 to provide multi-band light source outputs to illuminate asurgical site 80, including providing near-infrared or other non-visiblelight at predetermined intervals. At processing multi-band video signalsstep 256 shown in FIG. 7, the controller 34 processes video imagesignals that include the periodic reflected IR signals received by thevideo camera 64. Thus, step 250 essentially corresponds to step 150shown in FIG. 4, except the additional periodic IR signals are alsoprocessed.

The light source control image analysis subroutine 240 then is advancedby the controller 34 to step 252 shown in FIG. 9. At step 252, thecontroller 34 identifies the presence of bleeders in view of the IRsignals sensed by the video camera 64 in comparison with correspondingnon-visible video signatures for a video signal. Thus, step 252corresponds in part to step 152 shown in FIG. 4, except IR signals areprocessed instead of, or in addition to, visible color signals.

If a dispersion area is found at identify fast dispersion areas decisionstep 252, the controller 34 advances to stop identifier (ID) step 254.At step 254, the controller 34 stores an identifier corresponding to thepresence of bleeders. Thus, step 254 is essentially identical to step154 shown in FIG. 4.

Regardless of the identification of a bleeder, the light source controlimage analysis subroutine 240 shown in FIG. 9 advances to step 156 asshown in the subroutine 92 of FIG. 4.

In conclusion, the multiband light source control image analysissubroutine 240 corresponds to and replaces flowchart blocks 150, 152,154 shown in FIG. 4. The main difference between the flow chart shown inFIG. 9 from the corresponding portion of the flow chart illustrated inFIG. 4, is the use of IR light to determine the presence of dispersionareas, rather than color analysis.

Leak/Clog Detection Subroutine

In another embodiment of the invention, a leak/clog detection subroutine260 shown in FIG. 10 is also provided for use with the routine 86 shownin FIG. 3. The subroutine 260 operates to determine the presence ofgreater than expected fluid leakage at the operating site 80 or greaterthan expected fluid pressure in the flow paths.

In FIG. 10, the subroutine 260 begins at start 262 and advances to step264. At pressure comparison step 264 the input pressure, typicallydetermined from the pump system 50 or from a sensor associated withirrigation tubing 76 is measured and compared with an output pressure(suction) determined by a pressure sensor provided with a suction portof pump system 50 or otherwise pressure sensors disposed in suction flowpaths of tubing 52, 57 or within the cutting tool handpiece 44. When thesystem controller 34 determines that the pressure input and pressureoutput differ by a predetermined amount, the subroutine 260 advances toleak/clog store identifier (ID) step 266 and stores a clog identifier inthe system controller 34. Thereafter, whether or not a clog isidentified, the system controller 34 advances the subroutine 260 todecision step 268.

At flow decision step 268, the system controller 34 determines whetherthe measured fluid flow through the irrigation tubing 76 that enters thecannula 67 between the inner wall thereof and the periphery of thecamera/light source handpiece 66 and advances to the distal end of thecannula and enters the peritoneal cavity exceeds the fluid flow throughthe suction tubing 52, 57 by a significant amount. If the flowdifference is significant enough to determine that an issue exists atthe surgical site 80, such as significant leakage of fluid into the bodyof a patient, a leak identifier is stored in the system controller 34 atstep 270. Whether or not a leak is discovered at step 268, the routine260 advances to return step 274 as shown in FIG. 10.

As a result of the sensing of a clog, the fluid input and/or the fluidoutput can be automatically pulsed or varied by the controller 34.Further, the driving of the cutting tool 42 can be automaticallypulsed/stopped by the controller 34 in an attempt to unclog the suctionpath of the handpiece.

In some embodiments, the leak/clog detection subroutine 260 is operatedperiodically by the system controller 34 as a part of a routine that isdifferent from the routine 86 illustrated in FIG. 3.

Although a particular preferred embodiment of the invention has beendisclosed in detail for illustrative purposes, it will be recognizedthat variations or modifications of the disclosed apparatus, includingthe rearrangement of parts, lie within the scope of the presentinvention.

1. A video based image detection/identification system for fluid and visualization control of a surgical system for providing an acceptable video image of a surgical site, comprising: a cutting tool for manipulating tissue at a surgical site; a pump system for providing fluid to a surgical site; a light source for providing light to a surgical site; a video sensing device for obtaining video signals of images at the surgical site; an image display for displaying a video image; and a system controller capable of controlling at least one of the cutting tool, the pump system, and the video sensing device to maintain quality of video images obtained by the video sensing device and provided to the image display, wherein said controller receives and processes the video signals to determine a video signature corresponding to a condition that interferes with the quality of the video images, wherein said controller controls at least one of the cutting tool, the pump system, the light source and the video sensing device to address the condition of the fluid at the surgical site to return the video images from the video signals to an acceptable quality for viewing so that a user is free from having to manually control at least one of said cutting tool, said pump system, said light source, and said video sensing device to obtain an acceptable video image of a surgical site for viewing on the image display.
 2. The system according to claim 1, wherein the surgical system comprises an endoscopic system and the pump system comprises a liquid pump for providing liquid to a surgical site, and wherein the video signature to be determined corresponds to at least one of bubbles, debris, particles, and bleeders.
 3. The system according to claim 2, wherein the cutting tool comprises an RF cutting tool, and wherein the system controller controls the RF cutting tool in response to bubbles identified from the video signals by the system controller.
 4. The system according to claim 1, wherein the surgical system comprises a laparoscopic system and the pump system comprises an insufflator for providing gas to a surgical site, and wherein the video signature to be identified comprises at least one of smoke and debris, the system controller operating the pump to control the gas flow output in response to the amount of smoke or debris sensed.
 5. The system according to claim 1, wherein the system controller analyzes the video signal to determine a video signature corresponding to debris on a tip at a distal end of a housing of the video sensing device, the controller controlling a tip cleaning device in response to the sensing of debris at the tip of the housing.
 6. The system according to claim 1, wherein the system controller controls the light source to output predetermined wavelengths of energy at predetermined output levels for sensing preselected conditions.
 7. The system according to claim 6, wherein the predetermined wavelengths of energy are output intermittently.
 8. The system according to claim 6, wherein the predetermined wavelengths include near infrared energy for at least assisting in determining the presence of a bleeder or of blood in the video images of the video signals.
 9. The system according to claim 1, including controlling color balance of the video sensing device and light level output by the light source in order to obtain an acceptable video image.
 10. The system according to claim 1, wherein the system controller controls a suction pressure and/or a suction valve position of a cutting tool to obtain an acceptable video image.
 11. The system according to claim 1, wherein the system controller utilizes visible color image processing to determine the presence of blood within the video image.
 12. The system according to claim 1, wherein the color image processing comprises at least one of a FFT spatial frequency information technique and a weighting technique.
 13. A method for controlling a surgical system to provide an acceptable image of a surgical site for display, comprising: providing light energy to a surgical site so that a surgical site is viewable; obtaining video image signals at a surgical site with a video sensing device for use in displaying a video image on a display; analyzing the video image signals to determine the presence of factors that reduce visibility or clarity of the video image signals; in response to a lack of quality and clarity of the video image, controlling at least one of: a cutting tool, a pump system, a light source, and a video sensing device to affect fluid at a surgical site so that the image detected by the video signals is controlled to provide an acceptable image quality for viewing, without any manual control; and displaying the image on a video display.
 14. The method of claim 13, wherein a camera/light source handpiece includes the camera and the light source, the handpiece being disposed within a cannula that provides access to the surgical site, and the step of controlling at least one cutting tool, pump system, light source, and video sensing device comprises controlling the flow of irrigation fluid between the outer periphery of the camera/light source handpiece and an inner wall of the cannula, the fluid entering the surgical site via the distal end of the cannula.
 15. The method of claim 13, wherein the step of analyzing the video image signals includes comparing portions of the video image signals to video signatures corresponding to conditions that interfere with the quality of the visual signal.
 16. The method of claim 15, wherein the surgical system comprises an endoscopic system, and the video signatures are provided for bubbles, debris particles, bleeders, and other conditions.
 17. The method of claim 16, including the step of operating a cutting tool capable of affecting change in video image signals during operation thereof, wherein the step of analyzing the video image signals to determine the presence of factors that reduce visibility or clarity of the video image; and wherein the step of controlling to affect fluid at a surgical site in response to the video image signals comprises controlling suction of the cutting tool without manual input from a cutting tool operator.
 18. The method according to claim 13, including the step of automatically controlling at least one of light source intensity and of color balance of the video sensing device to enhance a video image for display without manual operation.
 19. The method according to claim 13, including providing light energy to the surgical site so that the surgical site is viewable for the video sensing device and providing non-visible light at one or more predetermined wavelengths for assisting the analyzing step in determining the presence of bleeders within the video image. 